Sensor information fusion device and method thereof

ABSTRACT

A sensor information fusion device and a method thereof are provided. The sensor information fusion device includes a processor that generates a first track box and a second track box based on an object detected by a plurality of sensors and determines whether the first track box and the second track box are overlapped with each other and a storage storing data obtained by the processor and an algorithm run by the processor. The processor generates a merge gate expanded from the first track box and determines the first track box and the second track box are overlapped with each other when the second track box is included in the merge gate.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of and priority to Korean PatentApplication No. 10-2020-0088352, filed on Jul. 16, 2020, the entirecontents of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to a sensor information fusion device anda method thereof, and more particularly, relates to technologies ofaccurately determining whether a plurality of sensor information areoverlapped with each other.

BACKGROUND

To implement smart cars of which humankind has dreamed, accuratepositioning of the vehicle, a precise positioning technology, a digitalmap technology, an advanced driving assistant system (ADAS) for checkinga driving state of the driver, and sensors and a sensor fusiontechnology used for the ADAS are very important.

The sensor fusion among them is a technology of recognizing situationsaround a host vehicle with high reliability through information fusionbased on sensor signals output from a front radar, a front view camera,a corner radar, and the like loaded into the host vehicle. Such a sensorfusion technology of the front radar and the front view camera isapplied to the ADAS, such as a smart cruise control (SCC) or a forwardcollision-avoidance assist (FCA), to be mass-produced.

Because information output by such sensors includes an overlappedportion, such an overlapped portion should be determined and deleted tofuse the information.

A method for obtaining a segment formula of a box capable of determiningmore various overlapped situations and determining whether there is anintersection between segments of two boxes has a high successful ratefor determining whether the two boxes are overlapped with each other.However, because the method needs the process of obtaining four pointsof a track box, obtaining a four-segment formula, and identifyingwhether there is an intersection between segments, a process fordetermining an overlapped situation is very complicated.

SUMMARY

An aspect of the present disclosure provides a sensor information fusiondevice for simply and accurately determining whether several tracks areoverlapped with each other to fuse the tracks when one object isdetected as the several overlapped tracks by a plurality of sensors anda method therefor.

The technical problems to be solved by the inventive concept are notlimited to the aforementioned problems, and any other technical problemsnot mentioned herein will be clearly understood from the followingdescription by those skilled in the art to which the present disclosurepertains.

According to an aspect of the present disclosure, a sensor informationfusion device may include a processor that generates a first track boxand a second track box based on an object detected by a plurality ofsensors and determine whether the first track box and the second trackbox are overlapped with each other and a storage storing data obtainedby the processor and an algorithm run by the processor. The processormay generate a merge gate expanded from the first track box anddetermine that the first track box and the second track box areoverlapped with each other when the second track box is included in themerge gate.

In an embodiment, the processor may calculate midpoints of the firsttrack box and the second track box based on information received fromthe plurality of sensors.

In an embodiment, the processor may convert the midpoint of the firsttrack box into the origin and may calculate coordinates of vertices ofthe first track box based on length information and width information ofthe first track box.

In an embodiment, the processor may convert the midpoint of the secondtrack box as relative coordinates for the origin of the first track boxby applying the midpoint of the first track box and the midpoint of thesecond track box to a rotation transformation formula.

In an embodiment, the processor may determine a size of the merge gatebased on length information and width information of the first trackbox.

In an embodiment, the processor may calculate coordinates of vertices ofthe second track box using the midpoint, length information, widthinformation, and a heading value of the second track box.

In an embodiment, the processor may determine whether the first trackbox and the second track box are overlapped with each other based on abox-in point function and may determine whether the first track box andthe second track box are overlapped with each other by applying a boxcrossed function to targets which are not determined by the box-in pointfunction.

In an embodiment, the processor may determine that the first track boxand the second track box are overlapped with each other, when at leastone of coordinates of vertices of the second track box is located in thefirst track box.

In an embodiment, the processor may divide a region outside the firsttrack box into a plurality of regions, when all of coordinates ofvertices of the second track box are not located in the first track box,and may determine whether the first track box and the second track boxare overlapped with each other based on a location of the coordinates,when the coordinates of the vertices of the second track box are locatedin the plurality of regions.

In an embodiment, the processor may sequentially divide and define aregion above the first track box as a first region, a second region, anda third region, may define a region at the left of the first track boxas a fourth region and defines a region at the right of the first trackbox as a fifth region, and may sequentially divide and define a regionbelow the first track box as a sixth region, a seventh region, and aneighth region.

In an embodiment, the processor may determine that the first track boxand the second track box are overlapped with each other, when at leastone of the coordinates of the vertices of the second track box ispresent in the second region and the seventh region or is present in thefourth region and the fifth region.

In an embodiment, the processor may form a first triangle by a lineextended after a first vertex of the second track box is connected witha first vertex of the first track box, a perpendicular line drawn fromthe first vertex of the second track box to the first track box, and aportion of a first surface of the first track box and may form a secondtriangle by a line extended after a second vertex of the second trackbox is connected with the first vertex of the first track box, aperpendicular line drawn from the second vertex of the second track boxto the first track box, and a portion of a second surface of the firsttrack box, when at least one of the coordinates of the vertices of thesecond track box is not present in the second region and the seventhregion and is not present in the fourth region and the fifth region.

In an embodiment, the processor may calculate a tangent value of thefirst triangle and a tangent value of the second triangle and maycompare the tangent value of the first triangle with the tangent valueof the second triangle.

In an embodiment, the processor may determine that the first track boxand the second track are overlapped with each other, when the tangentvalue of the first triangle is less than a tangent value of the secondtriangle.

In an embodiment, the processor may select a track box for fusiondepending on reliability of the first track box and reliability of thesecond track box.

In an embodiment, the processor may perform fusion using a track boxwith higher reliability, when the reliability of the first track box andthe reliability of the second track box have a difference of apredetermined reference value or more.

In an embodiment, the processor may select a track box, a generationtime of which is old, when the reliability of the first track box andthe reliability of the second track box have a difference of less than apredetermined reference value.

In an embodiment, the processor may calculate the reliability of thefirst track box and the reliability of the second track box usingreliability of each of sensors which provide information to generate thefirst track box and the second track box and a time when the first trackbox and the second track box are generated.

In an embodiment, the processor may delete a track box which is notselected for fusion.

According to another aspect of the present disclosure, a sensorinformation fusion method may include generating a first track box and asecond track box based on an object detected by a plurality of sensors,generating a merge gate expanded from the first track box anddetermining that the first track box and the second track box areoverlapped with each other, when the second track box is included in themerge gate, and deleting the first and second track boxes which areoverlapped with each other to perform fusion.

DRAWINGS

In order that the disclosure may be well understood, there will now bedescribed various forms thereof, given by way of example, referencebeing made to the accompanying drawings, in which:

FIGS. 1A, 1B, and 1C are drawings illustrating a conventional manner fordistinguishing overlapped track boxes;

FIG. 2 is a block diagram illustrating a configuration of a vehiclesystem including a sensor information fusion device in one form of thepresent disclosure;

FIG. 3 is a drawing illustrating an exemplary screen of convertingmidpoints of two track boxes in one form of the present disclosure;

FIG. 4 is a drawing illustrating an exemplary screen of converting asensor coordinate system into a reference coordinate system in one formof the present disclosure;

FIG. 5 is a drawing illustrating an exemplary screen of generating amerge gate in one form of the present disclosure;

FIGS. 6A and 6B are drawings illustrating exemplary screens of setting asize of a merge gate in one form of the present disclosure;

FIGS. 7A and 7B are drawings illustrating exemplary screens of applyinga box-in point function in one form of the present disclosure;

FIGS. 8A, 8B, 8C, and 8D are drawings illustrating exemplary screens ofapplying a box crossed function in one form of the present disclosure;

FIG. 9 is a flowchart illustrating a sensor information fusion method inone form of the present disclosure;

FIG. 10 is a flowchart illustrating in detail a method for determiningwhether a reference track and a target track are overlapped with eachother in one form of the present disclosure; and

FIG. 11 is a block diagram illustrating a computing system in one formof the present disclosure.

DETAILED DESCRIPTION

Hereinafter, some embodiments of the present disclosure will bedescribed in detail with reference to the exemplary drawings. In addingthe reference numerals to the components of each drawing, it should benoted that the identical or equivalent component is designated by theidentical numeral even when they are displayed on other drawings.Further, in describing the embodiment of the present disclosure, adetailed description of well-known features or functions will be ruledout in order not to unnecessarily obscure the gist of the presentdisclosure.

In describing the components of the embodiment according to the presentdisclosure, terms such as first, second, “A”, “B”, (a), (b), and thelike may be used. These terms are merely intended to distinguish onecomponent from another component, and the terms do not limit the nature,sequence or order of the constituent components. Unless otherwisedefined, all terms used herein, including technical or scientific terms,have the same meanings as those generally understood by those skilled inthe art to which the present disclosure pertains. Such terms as thosedefined in a generally used dictionary are to be interpreted as havingmeanings equal to the contextual meanings in the relevant field of art,and are not to be interpreted as having ideal or excessively formalmeanings unless clearly defined as having such in the presentapplication.

In an existing technology, to distinguish track boxes overlapped whenfusing sensors, a minimum (Min)/maximum (Max) manner is used as shown inFIG. 1A or a manner of calculating an intersection of track boxes isused as shown in FIG. 1C.

As shown in FIG. 1B, the Min/Max manner is applicable only when twoboxes are in a horizontal form. When directions of the two boxes differfrom each other, there may occur an error determination of whether thetwo boxes are overlapped with each other and all overlapped situationsare not determined.

Hereinafter, embodiments of the present disclosure will be described indetail with reference to FIGS. 2 to 11 .

FIG. 2 is a block diagram illustrating a configuration of a vehiclesystem including a sensor information fusion device according to anembodiment of the present disclosure.

A sensor information fusion device 100 according to an embodiment of thepresent disclosure may be implemented in a vehicle. In this case, thesensor information fusion device 100 may be integrally configured withcontrol units in the vehicle or may be implemented as a separate deviceto be connected with the control units of the vehicle by a separateconnection means.

Referring to FIG. 2 , the vehicle system may include the sensorinformation fusion device 100 and a sensing device 200.

The sensor information fusion device 100 may generate a first track box(a reference track) and a second track box (a target track) based on anobject detected by a plurality of sensors and may determine whether thefirst track box and the second track box are overlapped with each other.Particularly, the sensor information fusion device 100 may generate amerge gate expanded from the first track box and may determine that thefirst track box and the second track box are overlapped with each otherwhen the second track box is included in the merge gate.

For example, the sensor information fusion device 100 may receive sensorinformation respectively sensed by sensor A, sensor B, sensor C, andsensor D, may generate the reference track based on information receivedfrom sensor A and sensor B, and may generate the target track based oninformation received from sensor C and sensor D.

In this case, the reference track and the target track may beinformation about one overlapped object, but may be differentlyrecognized according to sensors. Thus, the sensor information fusiondevice 100 according to an embodiment of the present disclosure maydetect and delete such overlapped information to fuse sensorinformation.

The sensor information fusion device 100 according to an embodiment ofthe present disclosure, which performs the above-mentioned operations,may be implemented in the form of independent hardware including amemory and a processor for processing each operation or may be driven inthe form of being included in another hardware device such as amicroprocessor or a universal computer system.

To this end, the sensor information fusion device 100 may include acommunication device 110, a storage 120, an interface 130, and aprocessor 140.

The communication device 110 may be a hardware device implemented withvarious electronic circuits to transmit and receive a signal through awireless or wired connection. In an embodiment of the presentdisclosure, the communication device 110 may perform a networkcommunication technology in the vehicle and may performvehicle-to-infrastructure (V2I) communication with a server, aninfrastructure, or another vehicle outside the vehicle using wirelessInternet technology or short range communication technology. Herein, thenetwork communication technology in the vehicle may be to performinter-vehicle communication through controller area network (CAN)communication, local interconnect network (LIN) communication, flex-raycommunication, or the like. Furthermore, the wireless Internettechnology may include wireless local area network (WLAN), wirelessbroadband (WiBro), wireless-fidelity (Wi-Fi), world interoperability formicrowave access (WiMAX), or the like. Furthermore, the short rangecommunication technology may include Bluetooth, ZigBee, ultra wideband(UWB), radio frequency identification (RFID), infrared data association(IrDA), or the like.

As an example, the communication device 110 may receive results sensedby sensors in the sensing device 200.

The storage 120 may store a sensing result of the sensing device 200,data obtained by the processor 140, or data, algorithms, and/or the likenecessary for an operation of processor 140.

As an example, the storage 120 may store forward sensing information orthe like obtained by a camera, a light detection and ranging (LiDAR), aradar, or the like and may store a value preset by an experimental valueto determine a size of a merge gate.

The storage 120 may include at least one type of storage medium, such asa flash memory type memory, a hard disk type memory, a micro typememory, a card type memory (e.g., a secure digital (SD) card or anextreme digital (XD) card), a random access memory (RAM), a static RAM(SRAM), a read-only memory (ROM), a programmable ROM (PROM), anelectrically erasable PROM (EEPROM), a magnetic RAM (MRAM), a magneticdisk, and an optical disk.

The interface 130 may include an input means for receiving a controlcommand from a user and an output means for outputting an operationstate, an operation result, or the like of the sensor information fusiondevice 100. Herein, the input means may include a key button and mayfurther include a mouse, a joystick, a jog shuttle, a stylus pen, or thelike. Furthermore, the input means may further include a soft keyimplemented on a display.

The output means may include the display and a voice output means suchas a speaker. In this case, when a touch sensor such as a touch film, atouch sheet, or a touch pad is provided in the display, the displayoperates as a touchscreen and may be implemented in a form where theinput means and the output means are integrated with each other. As anexample, the output means may output information sensed by the sensingdevice 200 or may output sensor information fused by the sensorinformation fusion device 100.

In this case, the display may include at least one of a liquid crystaldisplay (LCD), a thin film transistor-LCD (TFT-LCD), an organiclight-emitting diode (OLED) display, a flexible display, a fieldemission display (FED), or a three-dimensional (3D) display.

The processor 140 may be electrically connected with the communicationdevice 110, the storage 120, the interface 130, or the like and mayelectrically control the respective components. The processor 140 may bean electrical circuit which executes instructions of software and mayperform a variety of data processing and calculation described below.The processor 140 may be, for example, an electronic control unit (ECU),a micro controller unit (MCU), or another sub-controller, which isloaded into the vehicle.

The processor 140 may generate the reference track (the first track box)and the target track (the second track box) for an object using sensingresults received from a plurality of sensors.

The processor 140 may calculate midpoints of the reference track and thetarget track based on information received from the plurality ofsensors. Referring to FIG. 3 , the processor 140 may generate trackboxes 10 and 20 for a detected object based on information about theobject detected by the plurality of sensors. In this case, as shown inFIG. 3 , a track box for one object may be generated differently foreach sensor. FIG. 3 is a drawing illustrating an exemplary screen ofconverting midpoints of two track boxes according to an embodiment ofthe present disclosure.

A processor 140 of FIG. 2 may generate midpoints (x1, y1) and (x2, y2)of track boxes 10 and 20.

In other words, the processor 140 may receive information about anobject detected by the plurality of sensors. The plurality of sensorsmay provide front midpoint information of the vehicle or rear midpointinformation of the vehicle as midpoint information of the detectedobject. Thus, the processor 140 may unify the front midpoint informationof the vehicle or the rear midpoint information of the vehicle into amidpoint of the vehicle to determine whether the track boxes 10 and 20are overlapped with each other. This is because the accuracy ofdetermining whether track boxes are overlapped with each other maybecome low when midpoint information of the track box differs frommidpoint information of the track box. In this case, the track boxes maybe divided into the reference track 10 and the target track 20.

The processor 140 may convert a midpoint of the reference track 10 intothe origin and may calculate coordinates of vertices of the referencetrack 10 based on length information and width information of thereference track 10. Referring to FIG. 4 , the processor 140 may convertthe midpoint of the reference track 10 into the origin (0, 0) and maycalculate coordinates of four vertices of the reference track 10 usinglength information L and width information W of the reference track 10.FIG. 4 is a drawing illustrating an exemplary screen of converting asensor coordinate system into a reference coordinate system according toan embodiment of the present disclosure. In FIG. 4 , coordinates of theleft upper vertex are (L/2, −W/2), coordinates of the right upper vertexare (L/2, W/2), coordinates of the left lower vertex are (−L/2, −W/2),and coordinates of the right lower vertex are (−L/2, W/2).

The processor 140 may apply the midpoint (x1, y1) of a reference track10 and the midpoint (x2, y2) of the target track 20 to a rotationtransformation formula in Equation 1 below to calculate the convertedmidpoint (x3, y3) of the target track 20. In other words, the processor140 may convert the midpoint (x1, y1) of the reference track 10 into themidpoint (0, 0) and may convert the midpoint (x2, y2) of the targettrack 20 into the midpoint (x3, y3) depending on the converted midpoint(0, 0) of the reference track 10.x3=(x2−x1)cos(−θ)−(y2−y1)sin(−θ)y3=(x2−x1)sin(−θ)+(y2−y1)cos(−θ)  [Equation 1]

The processor 140 may apply the midpoint of a reference track and themidpoint of a target track to the rotation transformation formula toconvert the midpoint of the target track as relative coordinates for theorigin of the reference track.

The processor 140 may generate a merge gate expanded from the referencetrack and may determine that the reference track and the target trackare overlapped with each other when the target track is included in themerge gate. Referring to FIG. 5 , when converting the midpoint as shownin FIG. 4 , the processor 140 may generate a merge gate 30 around thereference track 10. FIG. 5 is a drawing illustrating an exemplary screenof generating a merge gate according to an embodiment of the presentdisclosure. FIGS. 6A and 6B are drawings illustrating exemplary screensof setting a size of a merge gate according to an embodiment of thepresent disclosure. A processor 140 of FIG. 2 may set a horizontallength M and a vertical length N of the merge gate, which are preset byan experimental value, as shown in FIG. 6A, or may set a size of themerge gate using a length L and a width W of a reference track 10. Inother words, as shown in FIG. 6B, the processor 140 may add the width Wand the length L of the reference track 10 to the horizontal length Mand the vertical length N of the merge gate, which are set by theexperimental value, respectively, to determine a size of the merge gate.

In other words, the processor 140 may increase the length L and thewidth W of the reference track 10 at a predetermined rate to determine asize of the merge gate.

The processor 140 may calculate coordinates of vertices of a targettrack 20 using a midpoint, length information, width information, and aheading value of the target track 20.

The processor 140 may determine whether the reference track 10 and thetarget track 20 are overlapped with each other based on a box-in pointfunction and may apply a box crossed function to targets which aredetermined by the box-in point function to determine whether thereference track 10 and the target track 20 are overlapped with eachother.

In other words, the processor 140 may primarily determine whether thereference track 10 and the target track 20 are mostly overlapped witheach other using the box-in point function which needs a littlecalculation to minimize unnecessary calculation. When at least one ofcoordinates of vertices of the target track 20 is located in thereference track 10, the processor 140 may determine that the referencetrack 10 and the target track 20 are overlapped with each other.

FIGS. 7A and 7B are drawings illustrating exemplary screens of applyinga box-in point function to determine whether a reference track and atarget track are overlapped with each other according to an embodimentof the present disclosure.

When it is determined that at least one of vertices of a target track 20is included in a reference track 10 using coordinates of four verticesof the reference track 10 and coordinates of four vertices of the targettrack 20, a processor 140 of FIG. 2 may determine that the referencetrack 10 and the target track 20 are overlapped with each other.

The processor 140 may determine whether at least one of the vertices ofthe target track 20 is included in the reference track 10 using Equation2 below.(−L/2<x4<L/2)&&(−W/2<y5<W/2)  [Equation 2]

FIG. 7A illustrates an example where one vertex of the target track 20is located in the reference track 10. The processor 140 may determineEquation 2 above to determine whether coordinates (x4, y4) of one vertexof the target track 20 is present in the reference track 10.

In other words, the processor 140 may determine whether an x-axiscoordinate x4 is present in an x-axis length of the reference track 10and may determine whether a y-axis coordinate y4 is present in a y-axislength of the reference track 10, thus determining the coordinates (x4,y4) of one vertex of the target track 20 are present in the referencetrack 10. Meanwhile, FIG. 7B illustrates an example where all of fourvertices of the target track 20 are located in the reference track 10.

The processor 140 may secondarily apply a box crossed function totargets which are not determined using a box-in point function toadditionally determine whether the reference track 10 and the targettrack 20 are overlapped with each other. As such, the processor 140 mayfilter targets to be overlapped, once more by determining a location ofa point of the track box.

In other words, when all of coordinates of vertices of the target track20 are not located in the reference track 10, the processor 140 maydivide a region outside the reference track into a plurality of regions.When the coordinates of the vertices of the target track 20 are locatedin the plurality of regions, the processor 140 may determine whether thereference track 10 and the target track 20 are overlapped with eachother, depending on the location of the coordinates.

The processor 140 may sequentially divide and define a region above thereference track 10 as a first region, a second region, and a thirdregion, may define a region at the left of the reference track 10 as afourth region, may define a region at the right of the reference track10 as a fifth region, and may sequentially divide and define a regionbelow the reference track 10 as a sixth region, a seventh region, and aneighth region.

When at least one of coordinates of vertices of the target track 20 ispresent in the second region and the seventh region or in the fourthregion and the fifth region, the processor 140 may determine that thereference track 10 and the target track 20 are overlapped with eachother.

A description will be given in detail of an example of applying a boxcrossed function according to an embodiment of the present disclosureusing FIGS. 8A to 8D. FIGS. 8A to 8D are drawings illustrating exemplaryscreens of applying a box crossed function according to an embodiment ofthe present disclosure.

As shown in FIG. 8A, to detect that a target track 20 is overlapped witha reference track 10 although all of four vertices of the target track20 are not included in the reference track 10, a processor 140 maydivide a region outside the reference track 10 into eight regions{circle around (1)}, {circle around (2)}, {circle around (3)}, {circlearound (4)}, {circle around (5)}, {circle around (6)}, {circle around(7)}, and {circle around (8)}.

The processor 140 may determine whether at least one of coordinates offour vertices of the target track 20 is included in any of the eightregions outside the reference track 10 by comparing coordinates. In FIG.8A, it may be seen that the left upper vertex of the target track 20 islocated in region {circle around (2)}, that the right upper vertex islocated in region {circle around (3)}, that the left lower vertex islocated in region {circle around (7)}, and that the right lower vertexis located in region {circle around (8)}. As such, when four vertices ofthe target track 20 is located in the region outside the reference track10, the processor 140 may determine that the reference track 10 and thetarget track 20 are overlapped with each other.

FIG. 8B illustrates the case where four vertices of the target track 20are located in regions {circle around (2)}, {circle around (3)}, {circlearound (7)}, and {circle around (8)}. FIG. 8C illustrates the case wherefour vertices of the target track 20 are located in regions {circlearound (4)} and {circle around (5)}.

On the other hand, when at least one of coordinates of the vertices ofthe target track 20 is not present in the second region and the seventhregion and is not present in the fourth region and the fifth region, theprocessor 140 may form a first triangle by a line extended after a firstvertex of the target track 20 is connected with a first vertex of thereference track 10, a perpendicular line drawn from the first vertex ofthe target track 20 to the reference track 10, and a portion of a firstsurface of the reference track 10, and may form a second triangle by aline extended after a second vertex of the target track 20 is connectedwith the first vertex of the reference track 10, a perpendicular linedrawn from the second vertex of the target track 20 to the referencetrack 10, and a portion of a second surface of the reference track 10.

The processor 140 may calculate a tangent value of the first triangleand a tangent value of the second triangle to compare the tangent valueof the first triangle with the tangent value of the second triangle.When the tangent value of the first triangle is less than the tangentvalue of the second triangle, the processor 140 may determine that thereference track 10 and the target track 20 are overlapped with eachother.

Thus, the processor 140 may calculate a tan value for angles θ1, and θ2formed by the reference track 10 using the coordinates of the verticesof the target track 20 in FIG. 8D like Equation 3 below to calculate alength of the height/base, using the coordinates of the four vertices ofeach of the reference track 10 and the target track 20, which arepreviously calculated.

In other words, the processor 140 may know the height and base of eachof triangles 11 and 12 using the coordinates of the vertices of thetarget track 20 and may calculate tan □ by applying a length of theheight and base like Equation 3 below.tan Θ=Length of height/Length of base  [Equation 3]

The processor 140 may calculate tan θ1 of the triangle 11 of region{circle around (2)} and tan θ2 of the triangle 12 of region {circlearound (5)} and may compare tan θ1 with tan θ2. In other words, when thetan θ1 of the triangle 11 of region {circle around (2)} is less than tanθ2 of the triangle 12 of region {circle around (5)}, the reference track10 and the target track 20 may be overlapped with each other.

When the target box is located like reference numeral 40, that is, whentan θ1 of the triangle 11 of region {circle around (2)} is greater thanor equal to tan θ2 of the triangle 12 of region {circle around (5)},vertices of the reference track 10 are not present in the target track40 and thus the reference track 10 and the target track 20 are notoverlapped with each other.

Thus, the processor 140 may determine whether the reference track 10 isoverlapped with the target track 20 in the same method in all thedirections of four vertices of the reference track 10.

As such, when it is determined whether the reference track 10 and thetarget track 20 are overlapped with each other, the processor 140 mayselect a track box for fusion depending on reliability of each of thereference track 10 and the target track 20. The processor 140 maycalculate reliability of the reference track 10 and reliability of thetarget track 20 using reliability of each of sensors, which provideinformation to generate the reference track 10 and the target track 20,and a time when the reference track 10 and the target track 20 aregenerated.

In this case, when the reliability of the reference track 10 and thereliability of the target track 20 have a difference of a predeterminedvalue or more, the processor 140 may fuse track information using atrack box with high reliability and may delete a track box which is notselected for fusion, or may control to fuse track information and outputthe fused track information via an interface 130 of FIG. 2 .

When the reliability of the reference track 10 and the reliability ofthe target track 20 have a difference of less than the predeterminedreference value, the processor 140 may select a track box, a generationtime of which is old. Furthermore, when the reliability of the referencetrack 10 and the reliability of the target track 20 have a difference ofless than the predetermined reference value, that is, when thedifference between the reliability of the reference track 10 and thereliability of the target track 20 is low, the processor 140 may selecta track box using various conditions, such as a combination of tracks bysensors, a host vehicle reference distance, and a location of the track,other than a time when the track box is generated. As an example, whenthe difference between the reliability of the reference track 10 and thereliability of the target track 20 is low, the processor 140 may selecta track with many sensors which provide information about each track boxbetween the reference track 10 and the target track 20, may select atrack close to the host vehicle with respect to the host vehicle, or mayselect a track included in the line on which the host vehicle istraveling.

As such, the processor 140 may select a track to be deleted usingreliability of each of the reference track 10 and the target track 20and a time when the reference track 10 and the target track 20 aregenerated and may fuse information of the reference track 10 and thetarget track 20 by assigning a higher weight to information of a trackhaving high reliability on the basis of the reliability of each of thereference track 10 and the target track 20 and a time when the referencetrack 10 and the target track 20 are generated.

The processor 140 may calculate reliability of each track usingreliability of each sensor, which is received from each sensor, and anage when the track is generated, like Equation 4 below.Reliability of reference track(max 100)=α(Reliability of sensorA)+Reliability of sensor B)+r(Generation time of reference track)Reliability of target track=α(Reliability of sensor A)+β(Reliability ofsensor B)+r(Generation time of target track)  [Equation 4]

Like Equation 5 below, when a difference value between the reliabilityof the reference track 10 and the reliability of the target track 20 isgreater than a predetermined reference value f, the processor 140 mayselect and fuse a track with high reliability. When the difference valuebetween the reliability of the reference track 10 and the reliability ofthe target track 20 is less than the predetermined reference value f,the processor 140 may select and fuse a track, a generation time ofwhich is old.(reliability of reference track−Reliability of target track)>f%

Select track with high reliability(reliability of reference track−Reliability of target track)<f%

Select track,generation time of which is old  [Equation 5]

Herein, Equations 4 and 5 are described as being one example forselecting a track for fusion, but not limited thereto. The processor 140may select a track to be fused using various methods.

The processor 140 may control to delete and output a track, which haslow reliability because of being determined as being overlapped, fromthe result of the sensor fusion output.

The sensor device 200 may include one or more sensors which detect anobstacle, for example, a preceding vehicle, located around the vehicleand measure a distance from the obstacle and/or a relative speed of theobstacle.

The sensing device 200 may have a plurality of sensors for sensingobjects outside the vehicle and may obtain information about a locationof the object, a speed of the object, a movement direction of theobject, and/or a type (e.g., a vehicle, a pedestrian, a bicycle, amotorcycle, or the like) of the object. To this end, the sensing device200 may include an ultrasonic sensor, a radar, a camera, a laser scannerand/or a corner radar, a light detection and ranging (LiDAR), anacceleration sensor, a yaw rate sensor, a torque sensor and/or a wheelspeed sensor, a steering angle sensor, or the like.

The sensor information fusion device 100 according to an embodiment ofthe present disclosure, which has the above-mentioned configuration, isapplicable to an advanced driving assistant system (ADAS).

A description will be given in detail of a sensor information fusionmethod according to an embodiment of the present disclosure withreference to FIGS. 9 and 10 . FIG. 9 is a flowchart illustrating asensor information fusion method according to an embodiment of thepresent disclosure. FIG. 10 is a flowchart illustrating in detail amethod for determining whether a reference track and a target track areoverlapped with each other according to an embodiment of the presentdisclosure.

Hereinafter, it is assumed that an autonomous controller 100 of FIG. 2performs a process of FIGS. 9 and 10 . Furthermore, in a description ofFIGS. 9 and 10 , an operation described as being performed by anapparatus may be understood as being controlled by a processor 140 ofthe sensor information fusion device 100.

Referring to FIG. 9 , in S100, the apparatus may calculate midpoints ofa reference track 10 and a target track 20 based on information receivedfrom each sensor.

In S200, the apparatus may convert the midpoint of the reference track10 into the origin (0, 0) and may calculate coordinates of vertices ofthe reference track 10 based on length information and widthinformation.

In S300, the apparatus may calculate a location (midpoint) of the targettrack 20 based on a relative distance based on the converted midpoint(0, 0) of the reference track 10 and coordinates of each vertex.

In S400, the apparatus may determine whether the midpoint of the targettrack 20 is located in a merge gate 30. When the midpoint of the targettrack 20 is located in the merge gate 30, in S500, the apparatus maycalculate coordinates of each vertex of the target track 20.

In S600, the apparatus may determine whether the target track 20 and thereference track 10 are overlapped with each other. In this case, theapparatus may determine whether one of the coordinates of the verticesof the target track 20 is present in the reference track 10 (a box-inpoint) or may determine whether coordinates of the vertices of thetarget track 20 are present in a region outside the reference track 10(a box crossed point), thus determining whether the reference track 10and the target track 20 are overlapped with each other.

When it is determined that the reference track 10 and the target track20 are overlapped with each other, in S700, the apparatus may performfusion on the basis of a track with high reliability or a track, ageneration time of which is old, based on reliability of each of thereference track 10 and the target track 20.

In S800, the apparatus may delete the track, fused because of beingdetermined as being overlapped, from the result of the sensor fusionoutput.

Hereinafter, a description will be given in detail of the process (S600)of determining whether the reference track 10 and the target track 20are overlapped with each other with reference to FIG. 10 .

Referring to FIG. 10 , in S601, the apparatus may apply a box-in pointfunction. In S602 (in point), the apparatus may determine whether atleast one vertex of the target track 20 is located in the referencetrack 10.

When the at least one vertex of the target track 20 is located in thereference track 10, the apparatus may determine that the reference track10 and the target track 20 are overlapped with each other. When the atleast one vertex of the target track is not located in the referencetrack 10, in S603, the apparatus may apply a box crossed function. Whenthe at least one vertex of the target track 20 is not located in thereference track 10, but is located in a region outside the referencetrack 10, in S604 (Cross), the apparatus may determine that thereference track 10 and the target track 20 are overlapped with eachother. The apparatus may determine whether the reference track and thetarget track cross.

As such, an embodiment of the present disclosure may simply andaccurately determine whether two tracks are overlapped with each otherwhen determining whether one target information by a sensor isoverlapped and may accurately determine whether two tracks areoverlapped with each other although track directions between sensorsdiffer from each other.

Furthermore, an embodiment of the present disclosure may accuratelydetermine whether two tracks are overlapped with each other irrespectiveof a heading angle of each of the two tracks and may determine whethertracks, determination of which is easy, are overlapped with each otherin advance because of determining whether two tracks are overlapped witheach other in an order where a determination method is easy, thusreducing an unnecessary process.

Furthermore, an embodiment of the present disclosure may determine anoverlapped level in most of general situations using a resource similarto a logic which may correspond to only an existing special situation toperform fusion and may apply a software logic to improve sensor fusionperformance without increasing an additional material cost.

FIG. 11 is a block diagram illustrating a computing system according toan embodiment of the present disclosure.

Referring to FIG. 11 , a computing system 1000 may include at least oneprocessor 1100, a memory 1300, a user interface input device 1400, auser interface output device 1500, storage 1600, and a network interface1700, which are connected with each other via a bus 1200.

The processor 1100 may be a central processing unit (CPU) or asemiconductor device that processes instructions stored in the memory1300 and/or the storage 1600. The memory 1300 and the storage 1600 mayinclude various types of volatile or non-volatile storage media. Forexample, the memory 1300 may include a ROM (Read Only Memory) and a RAM(Random Access Memory).

Thus, the operations of the method or the algorithm described inconnection with the embodiments disclosed herein may be embodieddirectly in hardware or a software module executed by the processor1100, or in a combination thereof. The software module may reside on astorage medium (that is, the memory and/or the storage) such as a RAM, aflash memory, a ROM, an EPROM, an EEPROM, a register, a hard disk, aremovable disk, and a CD-ROM.

The exemplary storage medium may be coupled to the processor 1100, andthe processor 1100 may read information out of the storage medium andmay record information in the storage medium. Alternatively, the storagemedium may be integrated with the processor 1100. The processor and thestorage medium may reside in an application specific integrated circuit(ASIC). The ASIC may reside within a user terminal. In another case, theprocessor and the storage medium may reside in the user terminal asseparate components.

When one object is detected as several overlapped tracks by a pluralityof sensors, the present technology may simply and accurately determinewhether the several tracks are overlapped with each other to fuse thetracks.

In addition, various effects ascertained directly or indirectly throughthe present disclosure may be provided.

Hereinabove, although the present disclosure has been described withreference to exemplary embodiments and the accompanying drawings, thepresent disclosure is not limited thereto, but may be variously modifiedand altered by those skilled in the art to which the present disclosurepertains without departing from the spirit and scope of the presentdisclosure claimed in the following claims.

Therefore, the exemplary embodiments of the present disclosure areprovided to explain the spirit and scope of the present disclosure, butnot to limit them, so that the spirit and scope of the presentdisclosure is not limited by the embodiments. The scope of the presentdisclosure should be construed on the basis of the accompanying claims,and all the technical ideas within the scope equivalent to the claimsshould be included in the scope of the present disclosure.

What is claimed is:
 1. A sensor information fusion device, comprising: aprocessor configured to: when an object is detected by overlapping aplurality of track boxes through a plurality of sensors, generate afirst track box in which the object is detected by a first sensor amongthe plurality of sensors, and generate a second track box in which theobject is detected by a second sensor among the plurality of sensors;and determine whether the first track box and the second track box areoverlapped with each other; and a storage configured to store dataobtained by the processor and to store a set of instructions that whenexecuted by the processor, implement an algorithm, wherein the processoris further configured to execute the set of instructions stored in thestorage and to: generate a merge gate expanded from the first track box;and determine that the first track box and the second track box areoverlapped with each other when the second track box is included in themerge gate.
 2. The sensor information fusion device of claim 1, whereinthe processor is further configured to: calculate midpoints of the firsttrack box and the second track box based on information received fromthe plurality of sensors.
 3. The sensor information fusion device ofclaim 2, wherein the processor is further configured to: convert themidpoint of the first track box into an origin; and calculatecoordinates of vertices of the first track box based on lengthinformation and width information of the first track box.
 4. The sensorinformation fusion device of claim 3, wherein the processor is furtherconfigured to: convert the midpoint of the second track box as relativecoordinates for the origin of the first track box by applying themidpoint of the first track box and the midpoint of the second track boxto a rotation transformation formula.
 5. The sensor information fusiondevice of claim 3, wherein the processor is further configured to:calculate coordinates of vertices of the second track box using themidpoint, length information, width information, and a heading value ofthe second track box.
 6. The sensor information fusion device of claim1, wherein the processor is further configured to: determine a size ofthe merge gate based on length information and width information of thefirst track box.
 7. The sensor information fusion device of claim 1,wherein the processor is further configured to: determine whether thefirst track box and the second track box overlap with each other basedon a box-in point function; and determine whether the first track boxand the second track box overlap with each other by applying a boxcrossed function to targets which are not determined by the box-in pointfunction.
 8. The sensor information fusion device of claim 1, whereinthe processor is further configured to: determine that the first trackbox and the second track box overlap with each other when at least oneof coordinates of vertices of the second track box is located in thefirst track box.
 9. The sensor information fusion device of claim 1,wherein the processor is further configured to: divide a region outsidethe first track box into a plurality of regions when all of coordinatesof vertices of the second track box are not located in the first trackbox; and determine whether the first track box and the second track boxoverlap with each other based on a location of the coordinates when thecoordinates of the vertices of the second track box are located in theplurality of regions.
 10. The sensor information fusion device of claim9, wherein the processor is further configured to: sequentially divideand define a region above the first track box as a first region, asecond region, and a third region; define a region to left of the firsttrack box as a fourth region; define a region to right of the firsttrack box as a fifth region; and sequentially divide and define a regionbelow the first track box as a sixth region, a seventh region, and aneighth region.
 11. The sensor information fusion device of claim 10,wherein the processor is further configured to: determine that the firsttrack box and the second track box overlap with each other when at leastone of the coordinates of the vertices of the second track box ispresent in the second region and the seventh region or is present in thefourth region and the fifth region.
 12. The sensor information fusiondevice of claim 10, wherein the processor is further configured to: forma first triangle by a line extended after a first vertex of the secondtrack box is connected with a first vertex of the first track box, aperpendicular line drawn from the first vertex of the second track boxto the first track box, and a portion of a first surface of the firsttrack box; and form a second triangle by a line extended after a secondvertex of the second track box is connected with the first vertex of thefirst track box, a perpendicular line drawn from the second vertex ofthe second track box to the first track box, and a portion of a secondsurface of the first track box, when at least one of the coordinates ofthe vertices of the second track box is not present in the second regionand the seventh region and is not present in the fourth region and thefifth region.
 13. The sensor information fusion device of claim 12,wherein the processor is further configured to: calculate a tangentvalue of the first triangle and a tangent value of the second triangle;and compare the tangent value of the first triangle with the tangentvalue of the second triangle.
 14. The sensor information fusion deviceof claim 13, wherein the processor is further configured to: determinethat the first track box and the second track overlap with each otherwhen the tangent value of the first triangle is less than the tangentvalue of the second triangle.
 15. The sensor information fusion deviceof claim 1, wherein the processor is further configured to: select atrack box for fusion depending on reliability of the first track box andreliability of the second track box.
 16. The sensor information fusiondevice of claim 15, wherein the processor is further configured to:perform fusion using a track box with higher reliability when thereliability of the first track box and the reliability of the secondtrack box have a difference value equal to or greater than apredetermined reference value.
 17. The sensor information fusion deviceof claim 15, wherein the processor is further configured to: select atrack box having an old generation time when the reliability of thefirst track box and the reliability of the second track box have adifference value less than a predetermined reference value.
 18. Thesensor information fusion device of claim 15, wherein the processor isfurther configured to: calculate the reliability of the first track boxand the reliability of the second track box using reliability of each ofsensors; and generate the first track box and the second track box and atime when the first track box and the second track box are generated.19. The sensor information fusion device of claim 15, wherein theprocessor is further configured to: delete a track box which is notselected for fusion.
 20. A sensor information fusion method, comprising:when an object is detected by overlapping a plurality of track boxesthrough a plurality of sensors, generating a first track box in whichthe object is detected by a first sensor among the plurality of sensors,and a second track box in which the object is detected by a secondsensor among the plurality of sensors; generating a merge gate expandedfrom the first track box; and determining that the first track box andthe second track box are overlapped with each other when the secondtrack box is included in the merge gate.